Enterprises often have to process large sets of data and often have limited time in which to process that data. Regulatory organizations, such as federal and state governments, may require that these data sets are processed by certain deadlines and may impose fines and/or sanctions if the data sets are not processed by the required deadlines. Enterprises may employ microprocessors (e.g., central processing units (CPUs)) to process the data sets, however, because processing by microprocessors is not deterministic, the time required to process data sets by microprocessors is difficult to estimate. Accordingly, in order to ensure that deadlines are met, enterprises allocate more microprocessors than are necessary to process the data sets.
Further, because it is difficult to estimate how long microprocessors will take to process data sets, it is difficult to detect processing errors. Traditional systems may rely on a “timeout” approach to detect processing errors, where the processing time for the microprocessor is roughly estimated and applied to a long “timeout delay,” such that if the actual processing time exceeds the “timeout delay,” a processing error is assumed. However, because processing using microprocessors involves software processing, which is non-deterministic, it is difficult to estimate how long microprocessors will take to process data sets and these timeout delays are set to unreasonable large numbers. As such, in practice, timeout delays are often ignored.
Moreover, the processing requirements for data may vary and the processing capacity of processing resources may vary. Often, the processing requirements for a data set are not optimized for the processing capabilities of available processing resources. This can lead to inefficient allocation of processing resources, such as high processing capacity processing resources being allocated to data with low processing requirements while data with higher processing requirements is queued for processing because available processing resources, which could have processed the data allocated to the processing resources processing the data with low processing requirements, are unable to process the data with high processing requirements.